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Abstract 

The problem is sequence prediction in the following setting. A sequence x±, . . . , x n , . . . of 
discrete- valued observations is generated according to some unknown probabilistic law (measure) 
(i. After observing each outcome, it is required to give the conditional probabilities of the next 
observation. The measure /i belongs to an arbitrary but known class C of stochastic process 
measures. We are interested in predictors p whose conditional probabilities converge (in some 
sense) to the "true" ^-conditional probabilities if any fj, £ C is chosen to generate the sequence. 
The contribution of this work is in characterizing the families C for which such predictors exist, 
and in providing a specific and simple form in which to look for a solution. We show that if any 
predictor works, then there exists a Bayesian predictor, whose prior is discrete, and which works 
too. We also find several sufficient and necessary conditions for the existence of a predictor, in 
terms of topological characterizations of the family C, as well as in terms of local behaviour of 
the measures in C, which in some cases lead to procedures for constructing such predictors. 

It should be emphasized that the framework is completely general: the stochastic processes 
considered are not required to be i.i.d., stationary, or to belong to any parametric or countable 
family. 

1 Introduction 

Given a sequence xi,...,x n of observations Xi G X, where X is a finite set, we want to predict 
what are the probabilities of observing x n+ \ = x for each x £ X, or, more generally, probabilities 
of observing different x n+ i, . . . , x n+ h, before x n+ \ is revealed, after which the process continues. 
It is assumed that the sequence is generated by some unknown stochastic process fj,, a probability 
measure on the space of one-way infinite sequences X°°. The goal is to have a predictor whose 
predicted probabilities converge (in a certain sense) to the correct ones (that is, to /^-conditional 
probabilities). In general this goal is impossible to achieve if nothing is known about the measure 
[i generating the sequence. In other words, one cannot have a predictor whose error goes to zero 
for any measure fi. The problem becomes tractable if we assume that the measure fi generating 
the data belongs to some known class C. The questions addressed in this work are a part of the 
following general problem: given an arbitrary set C of measures, how can we find a predictor that 
performs well when the data is generated by any \i £ C, and whether it is possible to find such 
a predictor at all. An example of a generic property of a class C that allows for construction of 
a predictor, is that C is countable. Clearly, this condition is very strong. An example, important 
from the applications point of view, of a class C of measures for which predictors are known, is the 
class of all stationary measures. The general question, however, is very far from being answered. 

The contribution of this work to solving this question is, first, in that we provide a specific 
form in which to look for a predictor. More precisely, we show that if a predictor that predicts 
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every p E C exists, then such a predictor can also be obtained as a weighted sum of countably 
many elements of C. This result can also be viewed as a justification of the Bayesian approach to 
sequence prediction: if there exists a predictor which predicts well every measure in the class, then 
there exists a Bayesian predictor (with a rather simple prior) that has this property too. In this 
respect it is important to note that the result obtained about such a Bayesian predictor is pointwise 
(holds for every p in C), and stretches far beyond the set its prior is concentrated on. Next, we 
derive some characterizations of families C for which a predictor exist. We first analyze what is 
furnished by the notion of separability, when a suitable topology can be found: we find that it is a 
sufficient but not always a necessary condition. We then derive some sufficient conditions for the 
existence of a predictor which are based on local (truncated to the first n observation) behaviour of 
measures in the class C. Necessary conditions cannot be obtained in this way (as we demonstrate), 
but sufficient conditions, along with rates of convergence and construction of predictors, can be 
found. 

The motivation for studying predictors for arbitrary classes C of processes is two-fold. First 
of all, prediction is a basic ingredient for constructing intelligent systems. Indeed, in order to be 
able to find optimal behaviour in an unknown environment, an intelligent agent must be able, at 
the very least, to predict how the environment is going to behave (or, to be more precise, how 
relevant parts of the environment are going to behave). Since the response of the environment 
may in general depend on the actions of the agent, this response is necessarily non-stationary for 
explorative agents. Therefore, one cannot readily use prediction methods developed for stationary 
environments, but rather has to find predictors for the classes of processes that can appear as a 
possible response of the environment. 

Apart from this, the problem of prediction itself has numerous applications in such diverse 
fields as data compression, market analysis, bioinformatics, and many others. It seems clear that 
prediction methods constructed for one application cannot be expected to be optimal when applied 
to another. Therefore, an important question is how to develop specific prediction algorithms for 
each of the domains. 

Prior work. As it was mentioned, if the class C of measures is countable (that is, if C can be 
represented as C := {p^ : k E N}), then there exists a predictor which performs well for any p £ C. 
Such a predictor can be obtained as a Bayesian mixture ps := SfceN Wkpk-, where u>k are summable 
positive real weights, and it has very strong predictive properties; in particular, pg predicts every 
p G C in total variation distance, as follows from the result of [Blackwell and Dubins(1962)| . Total 
variation distance measures the difference in (predicted and true) conditional probabilities of all fu- 
ture events, that is, not only the probabilities of the next observations, but also of observations that 
are arbitrary far off in the future (see formal definitions below). In the context of sequence predic- 
tion the measure ps was first studied by |Solomonoff(1978)] . Since then, the idea of taking a convex 
combination of a finite or countable class of measures (or predictors) to obtain a predictor permeates 
most of the research on sequential prediction (see, for example, Cesa-Bianchi and Lugosi(2006) |) 



and more general learning problems |Hutter(2005) Ryabko and Hutter (2008a) . In practice it is 



clear that, on the one hand, countable models are not sufficient, since already the class p p ,p G [0, 1] 
of Bernoulli i.i.d. processes, where p is the probability of 0, is not countable. On the other hand, 
prediction in total variation can be too strong to require; predicting probabilities of the next obser- 
vation may be sufficient, maybe even not on every step but in the Cesaro sense. A key observation 
here is that a predictor ps = J2 w kPk may be a good predictor not only when the data is generated 
by one of the processes pk, k G N, but when it comes from a much larger class. Let us consider 
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this point in more detail. Fix for simplicity X = {0, 1}. The Laplace predictor 



\{x n+1 = ari, . . . ,x n ) = —— (1) 

n + \X\ 

predicts any Bernoulli i.i.d. process: although convergence in total variation distance of conditional 
probabilities does not hold, predicted probabilities of the next outcome converge to the correct ones. 
Moreover, generalizing the Laplace predictor, a predictor can be constructed for the class of 
all fc-order Markov measures, for any given k. As was found by |R yabko(1988)| , the combination 



Pr := ^2 w k^k is a good predictor not only for the set U^nM^ of all finite-memory processes, but 
also for any measure p coming from a much larger class: that of all stationary measures on X°° . 
Here prediction is possible only in the Cesaro sense (more precisely, pn predicts every stationary 
process in expected time-average Kullback-Leibler divergence, see definitions below). The Laplace 
predictor itself can be obtained as a Bayes mixture over all Bernoulli i.i.d. measures with uniform 
prior on the parameter p (the probability of 0). However, as was observed in |Hutter(2007)| (and 
as is easy to see), the same (asymptotic) predictive properties are possessed by a Bayes mixture 
with a countably supported prior which is dense in [0, 1] (e.g. taking p '■= w^Sk where 6 N 
ranges over all Bernoulli i.i.d. measures with rational probability of 0). For a given k, the set 
of fc-order Markov processes is parametrized by finitely many [0, l]-valued parameters. Taking 
a dense subset of the values of these parameters, and a mixture of the corresponding measures, 
results in a predictor for the class of fe-order Markov processes. Mixing over these (for all k £ N) 
yields, as in Ryabko(1988)| , a predictor for the class of all stationary processes. Thus, for the 



mentioned classes of processes, a predictor can be obtained as a Bayes mixture of countably many 
measures in the class. An additional reason why this kind of analysis is interesting is because of 
the difficulties arising in trying to construct Bayesian predictors for classes of processes that can 
not be easily parametrized. Indeed, a natural way to obtain a predictor for a class C of stochastic 
processes is to take a Bayesian mixture of the class. To do this, one needs to define the structure of a 
probability space on C. If the class C is well parametrized, as is the case with the set of all Bernoulli 
i.i.d. process, then one can integrate with respect to the parametrization. In general, when the 
problem lacks a natural parametrization, although one can define the structure of the probability 
space on the set of (all) stochastic process measures in many different ways, the results one can 
obtain will then be with probability 1 with respect to the prior distribution (see, for example, 
[Jackson et al. (1999) Jackson, Kalai, and Smorodinsky] ) . Pointwise consistency cannot be assured 



(see e.g. |Diaconis and Freedman(1986) ) in this case, meaning that some (well-defined) Bayesian 
predictors are not consistent on some (large) subset of C. Results with prior probability 1 can be 
hard to interpret if one is not sure that the structure of the probability space defined on the set C is 
indeed a natural one for the problem at hand (whereas if one does have a natural parametrization, 
then usually results for every value of the parameter can be obtained, as in the case with Bernoulli 
i.i.d. processes mentioned above). The results of the present work show that when a predictor 
exists it can indeed be given as a Bayesian predictor, which predicts every (and not almost every) 
measure in the class, while its support is only a countable set. 

A related question is formulated as a question about two individual measures, rather than about 
a class of measures and a predictor. Namely, one can ask under which conditions one stochastic 
process predicts another. In Blackwell and Dubins(1962)] it was shown that if one measure is abso- 



lutely continuous with respect to another, than the latter predicts the former (the conditional proba- 
bilities converge in a very strong sense) . In [Ryabko and Hutter(2007)||Ryabko and Hutter (2008b) | 
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a weaker form of convergence of probabilities (in particular, convergence of expected average KL 
divergence) is obtained under weaker assumptions. 

The results. First, we show that if there is a predictor that performs well for every measure 
coming from a class C of processes, then a predictor can also be obtained as a convex combination 
SfcgN w k^k for some p^ G C and some > 0, k G N. This holds if the prediction quality is 
measured by either total variation distance, or expected average KL divergence: one measure of 
performance that is very strong, the other rather weak. The analysis for the total variation case 
relies on the fact that if p predicts p in total variation distance, then p is absolutely continuous 
with respect to p, so that p{x\,, n ) / p{x\,, n ) converges to a positive number with /x-probability 1 and 
with a positive p-probability. However, if we settle for a weaker measure of performance, such as 
expected average KL divergence, measures p G C are typically singular with respect to a predictor 
p. Nevertheless, since p predicts p we can show that p{xi.. n ) / p(x\.. n ) decreases subexponentially 
with n (with high probability or in expectation); then we can use this ratio as an analogue of the 
density for each time step n, and find a convex combination of countably many measures from C 
that has desired predictive properties for each n. Combining these predictors for all n results in a 
predictor that predicts every p G C in average KL divergence. The proof techniques developed have 
a potential to be used in solving other questions concerning sequence prediction, in particular, the 
general question of how to find a predictor for an arbitrary class C of measures. 

We then exhibit some sufficient conditions on the class C, under which a predictor for all 
measures in C exists. It is important to note that none of these conditions relies on a parametrization 
of any kind. The conditions presented are of two types: conditions on asymptotic behaviour of 
measures in C, and on their local (restricted to first n observations) behaviour. Conditions of the 
first type concern separability of C with respect to the total variation distance and the expected 
average KL divergence. We show that in the case of total variation separability is a necessary and 
sufficient condition for the existence of a predictor, whereas in the case of expected average KL 
divergence it is sufficient but is not necessary. 

The conditions of the second kind concern the "capacity" of the sets C n := {p n : p G C}, n G N, 
where p n is the measure p restricted to the first n observations. Intuitively, if C n is small (in some 
sense), then prediction is possible. We measure the capacity of C n in two ways. The first way is 
to find the maximum probability given to each sequence x\, . . . ,x n by some measure in the class, 
and then take a sum over x±, . . . , x n . Denoting the obtained quantity c n , one can show that it 
grows polynomially in n for some important classes of processes, such as i.i.d. or Markov processes. 
We show that, in general, if c n grows subexponentially then a predictor exists that predicts any 
measure in C in expected average KL divergence. On the other hand, exponentially growing c n are 
not sufficient for prediction. A more refined way to measure the capacity of C n is using a concept 
of channel capacity from information theory, which was developed for a closely related problem 
of finding optimal codes for a class of sources. We extend corresponding results from information 
theory to show that sublinear growth of channel capacity is sufficient for the existence of a predictor, 
in the sense of expected average divergence. Moreover, the obtained bounds on the divergence are 
optimal up to an additive logarithmic term. 

The rest of the paper is organized as follows. Section [2] introduces the notation and definitions. 
In Section [3] we show that if any predictor works than there is a Bayesian one that works, while 
in Section [4] we provide several characterizations of predictable classes of processes. Section 14.11 
is concerned with separability, while Section 14.21 analyzes conditions based on local behaviour of 
measures. Finally, Section [5] provides outlook and discussion. 
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As running examples that illustrate the results of each section we use countable classes of 
measures, the family of all Bernoulli i.i.d. processes and that of all stationary processes. 



2 Preliminaries 

Let X be a finite set. The notation x\__ n is used for x±, . . . ,x n . We consider stochastic processes 
(probability measures) on (X°°, J 7 ) where T is the sigma-field generated by the cylinder sets [xi.. n ], 
Xi G X,n £ N, where [xi.. n ] is the set of all infinite sequences that start with x\_, n . For a finite set 
A denote |^4| its cardinality. We use E^ for expectation with respect to a measure p. 

Next we introduce the measures of the quality of prediction used in this paper. For two measures 
p and p we are interested in how different the p- and p-conditional probabilities are, given a data 
sample x\_, n . Introduce the (conditional) total variation distance 

v(jj,,p,xu n ) : = sup \p(A\xi.. n ) - p(A\xun)\- 

Definition 1. We say that p predicts p in total variation if 

v(p,p,xi.. n ) ->• p-a.s. 

This convergence is rather strong. In particular, it means that p-conditional probabilities of 
arbitrary far-off events converge to //-conditional probabilities. Moreover, p predicts p in total 
variation if |Blackwell and Dubins(196 2) | and only if Kalai and Lehrer(1994)| p is absolutely con- 
tinuous with respect to p: 



Theorem 1 ( [Blackwell and Dubins(1962)[ |Kalai and Lehrer(1994)| ). // p, p are arbitrary prob- 



ability measures on (X 00 ,^), then p predicts p in total variation if and only if p is absolutely 
continuous with respect to p. 

Thus, for a class C of measures there is a predictor p that predicts every p £ C in total variation 
if and only if every p £ C has a density with respect to p. Although such sets of processes are 
rather large, they do not include even such basic examples as the set of all Bernoulli i.i.d. processes. 
That is, there is no p that would predict in total variation every Bernoulli i.i.d. process measure 
S p , p 6 [0, 1], where p is the probability of 0. Therefore, perhaps for many (if not most) practical 
applications this measure of the quality of prediction is too strong, and one is interested in weaker 
measures of performance. 

For two measures p and p introduce the expected cumulative Kullback-Leibler divergence (KL 
divergence) as 

d n (p, p) := V V p(x t = a\xi..t-i) log ^ a \ x i..t-i) ^ 

In words, we take the expected (over data) average (over time) KL divergence between p- and 
p-conditional (on the past data) probability distributions of the next outcome. 

Definition 2. We say that p predicts p in expected average KL divergence if 

-d n (p,p) -> 0. 
n 
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This measure of performance is much weaker, in the sense that it requires good predictions 
only one step ahead, and not on every step but only on average; also the convergence is not with 
probability 1 but in expectation. With prediction quality so measured, predictors exist for relatively 
large classes of measures; most notably, Ryabko(1988)| provides a predictor which predicts every 



stationary process in expected average KL divergence. A simple but useful identity that we will 
need (in the context of sequence prediction introduced also in [Ryabko(198 8) |) is the following 

dn{^,P)=- Yl M(^l..n)log P , Xl " H s , (3) 

where on the right-hand side we have simply the KL divergence between measures p and p restricted 
to the first n observations. 

Thus, the results of this work will be established with respect to two very different measures 
of prediction quality, one of which is very strong and the other rather weak. This suggests that 
the facts established reflect some fundamental properties of the problem of prediction, rather than 
those pertinent to particular measures of performance. On the other hand, it remains open to 
extend the results below to different measures of performance. 

3 Fully nonparametric Bayes predictors 

In this section we show that if there is a predictor that predicts every /x in some class C, then 
there is a Bayesian mixture of countably many elements from C that predicts every /i E C too. 
This is established for the two notions of prediction quality that were introduced: total variation 
and expected average KL divergence. After the theorems we present some examples of families of 
measures for which predictors exist. 

Theorem 2. Let C be a set of probability measures on (X°° ^T). If there is a measure p such that 
p predicts every \x E C in total variation, then there is a sequence pk £ C, k E N such that the 
measure v := ^^^WkPk predicts every /x E C in total variation, where are any positive weights 
that sum to 1. 

This relatively simple fact can be proven in different ways, relying on the mentioned equiv- 
alence Blackwell and Dubins(1962) , Kalai and Lehrer(1994) | of the statements u p predicts p in 



total variation distance" and LL p is absolutely continuous with respect to p." The proof presented 
below is not the shortest possible, but it uses ideas and techniques that are then generalized to the 
case of prediction in expected average KL-divergence, which is more involved, since in all interest- 
ing cases all measures p E C are singular with respect to any predictor that predicts all of them. 
Another proof of Theorem [2] can be obtained from Theorem [5] in the next section. Yet another 
way would be to derive it from algebraic properties of the relation of absolute continuity, given in 
Plesner and Rokhlin(1946)] . 



Proof. We break the (relatively easy) proof of this theorem into 3 steps, which will make the proof 
of the next theorem more understandable. 

Step 1: densities. For any fi E C, since p predicts p in total variation, by Theorem [IJ p has a 
density (Radon-Nikodym derivative) /„ with respect to p. Thus, for the set of all sequences 
x\,X2, ... E X°° on which (2:1,2,...) > (the limit lim^oo exists and is finite and positive) 
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we have /x(T^) = 1 and p(T^) > 0. Next we will construct a sequence of measures /ij 6 C, I; 6 N 
such that the union of the sets Tu k has probability 1 with respect to every p £ C, and will show 
that this is a sequence of measures whose existence is asserted in the theorem statement. 



Step 2: a countable cover and the resulting predictor. Let e k := 2 and let mi := sup^gg p(T^ 



A' / 



Clearly, m\ > 0. Find any p\ £ C such that /o(T A11 ) > mi — ej_, and let T\ = T^. For k > 1 define 
mfc := sup MgC p(T^\Tk-\). If mfc = then define := T^-i, otherwise find any p k such that 
p(T Mfc \T fc _i) >m k - s k , and let T fe := T fc _i U T Mfc . Define the predictor v as f := Y^keN w k^k- 

Step 3: v predicts every p £ C. Since the sets Ti, T2\Ti, . . . , T k \T k _i, ■ ■ ■ are disjoint, we must 
have p(T k \T k _i) — ► 0, so that m k — ► (since m k < p(T k \T k _i) + — > 0). Let 

Fix any p £ C. Suppose that p(T^\T) > 0. Since \x is absolutely continuous with respect to p, we 
must have 5 := p(T^\T) > 0. Then for every k > 1 we have 

m fc = sup p(T M /\T fc _i) > p(Tfj\T k ^i) > p(T M \T) = 5 > 0, 
which contradicts m^ — > 0. Thus, we have shown that 

M(rnr M ) = i. (4) 

Let us show that every /i S C is absolutely continuous with respect to v. Indeed, fix any p £ C 
and suppose p(A) > for some A £ T . Then from (jl]) we have p(A n T) > 0, and, by absolute 
continuity of /i with respect to p, also /)(4nT) > 0. Since T = UkenTk we must have p(Ar\Tk) > 
for some k £ N. Since on the set T k the measure /U^ has non-zero density with respect to p, we 
must have Pk{A n T^,) > 0. (Indeed, n Tfc) = J AnT ffi k dp > 0.) Hence, 

v{AP\T k ) > w k p k (AnT k ) > 0, 

so that v(A) > 0. Thus, p is absolutely continuous with respect to u, and so, by Theorem [H v 
predicts p in total variation distance. □ 

Thus, examples of families C for which there is a p that predicts every p £ C in total variation, 
are limited to families of measures which have a density with respect to some measure p. On the 
one hand, from statistical point of view, such families are rather large: the assumption that the 
probabilistic law in question has a density with respect to some (nice) measure is a standard one 
in statistics. It should also be mentioned that such families can easily be uncountable. On the 
other hand, even such basic examples as the set of all Bernoulli i.i.d. measures does not allow for 
a predictor that predicts every measure in total variation. Indeed, all these processes are singular 
with respect to one another; in particular, each of the non-overlapping sets T p of all sequences 
which have limiting fraction p of 0s has probability 1 with respect to one of the measures and 
with respect to all others; since there are uncountably many of these measures, there is no measure p 
with respect to which they all would have a density (since such a measure should have p(T p ) > for 
all p) . As it was mentioned, predicting in total variation distance means predicting with arbitrarily 
growing horizon Kalai and Lehrer(1994)| , while prediction in expected average KL divergence is 



only concerned with the probabilities of the next observation, and only on time and data average. 
For the latter measure of prediction quality, consistent predictors exist not only for the class of 
all Bernoulli processes, but also for the class of all stationary processes [Ryabko(1988)1 . The next 



theorem establishes the result similar to Theorem [2] for expected average KL divergence. 



7 



Theorem 3. Let C be a set of probability measures on (X 00 ,^). If there is a measure p such that 
p predicts every p £ C in expected average KL divergence, then there exist a sequence pk £ C, k £ N 
and a sequence Wk > 0, k 6 N, such that J2k eN w k = 1? anc ^ ^ e measure v := YlkzN w kf J 'k predicts 
every p £ C in expected average KL divergence. 

A difference worth noting with respect to the formulation of Theorem [2] (apart from a different 
measure of divergence) is in that in the latter the weights can be chosen arbitrarily, while in 
Theorem[3]this is not the case. In general, the statement "X] fceN w k l/ k predicts p in expected average 
KL divergence for some choice of Wk, k £ N" does not imply "X^fceN w 'k v k P re dicts p in expected 
average KL divergence for every summable sequence of positive w' k ,k £ N," while the implication 
trivially holds true if the expected average KL divergence is replaced by the total variation. This 
is illustrated in the last example of this section. An interesting related question (which is beyond 
the scope of this paper) is how to chose the weights to optimize the behaviour of a predictor before 
asymptotic. 

The idea of the proof of Theorem [3] is as follows. For every p and every n we consider the 
sets T™ of those cci.. n on which p is greater than p. These sets have to have (from some n on) a 
high probability with respect to p. Then since p predicts p in expected average KL divergence, 
the ^-probability of these sets cannot decrease exponentially fast (that is, it has to be quite large). 
(The sequences p{xi,, n ) / p{x\,, n ), n £ N will play the role of densities of the proof of Theorem [21 
and the sets T™ the role of sets on which the density is non-zero.) We then use, for each given 
n, the same scheme to cover the set X n with countably many T™, as was used in the proof of 
Theorem [2] to construct a countable covering of the set X°° , obtaining for each n a predictor v n . 
Then the predictor v is obtained as X^ngN w n^n, where the weights decrease subexponentially. The 
latter fact ensures that, although the weights depend on n, they still play no role asymptotically. 
The technically most involved part of the proof is to show that the sets T™ in asymptotic have 
sufficiently large weights in those countable covers that we construct for each n. This is used 
to demonstrate the implication "if a set has a high p probability, then its ^-probability does not 
decrease too fast, provided some regularity conditions." The proof is broken into the same steps as 
the (simpler) proof of Theorem [2 to make the analogy explicit and the proof more understandable. 

Proof. Define the weights Wk := wk~ 2 , where w is the normalizer 6/n 2 . 
Step 1: densities. Define the sets 

T M n := £ X n : p( Xl .. n ) > ^p(xi.. n )\ • (5) 



Using Markov's inequality, we derive 



( p(xi..n) \ < 1 „ p(xi.. n ) 



1 



p(X n \Tp = p > n < -E,^^- = -, (6) 

^ \p{xi..n) ) n p p(xi„ n ) n 

so that p(T™) — * 1. (Note that if p is singular with respect to p, as is typically the case, then 
-fcj—^ converges to p-a.e. and one can replace ^ in ([5]) by 1, while still having p(T™) — > 1.) 

Step 2n: a countable cover, time n. Fix an n £ N. Define m™ := max^gc p(TJ}) (since X n are 
finite all suprema are reached). Find any p™ such that Pi(T™ n ) = m" and let T" := T%». For k > 1, 
let ml := max^c piT^T^). If m% > 0, let jtg be any p £ C such that p(T^\T£_J = mg, and 
let TJ} := T^_ x U T"„; otherwise let := T^_ v Observe that (for each n) there is only a finite 
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number of positive rriu, since the set X n is finite; let K n be the largest index k such that m£ > 0. 
Let 

i/„ :=5^tUfcMfc. (7) 

k=l 

As a result of this construction, for every re G N every < K n and every aci.. n G XJ™ using ([5]) we 
obtain 

fn(si..n) > W k -p(xi„ n ). (8) 

n 

5£ep £/te resulting predictor. Finally, define 



1 1 

^7 + ^2^ WnUn > ^> 



v : 

2 ' 2 

ngN 

where 7 is the i.i.d. measure with equal probabilities of all x G X (that is, 7(^1. , n ) = |<-f| _n for 
every re G N and every x\,_ n G X n ). We will show that v predicts every /i£C, and then in the end 
of the proof (Step r) we will show how to replace 7 by a combination of a countable set of elements 
of C (in fact, 7 is just a regularizer which ensures that z^-probability of any word is never too close 
to 0). 

Step 3: v predicts every p, G C. Fix any p £ C. Introduce the parameters e™ E (0,1), n £ 
N, to be defined later, and let j™ := 1/eJJ. Observe that p(T^\T^_ 1 ) > p(T" +1 \T fe n ), for any 
k > 1 and any n G N, by definition of these sets. Since the sets T^\T^_ 1 , k G N are disjoint, 
we obtain /9(T^\r^_ 1 ) < l//c. Hence, p(T™\T™) < for some j < j™, since otherwise = 
max^c p(T^\T^) > e™ so that p(Tj^ +1 \Tjp > e™ = which is a contradiction. Thus, 

p(t;\t%)<sI. (10) 

We can upper-bound p(T^\Tyi) as follows. First, observe that 
d n {V,p) = - ^2 M(ffl..n)log p xi " n 



xi..„ST™nT« 
p J/J 



^ ^(x L . n )log P "' 1 



u(xi 

xi..„eT«\T» 



1 " 



^ ^(xL.^log— — 



I + 11 + III. (11) 



Then, from J5]) we get 

I>-logn. (12) 
Observe that for every re G N and every set A C A" n , using Jensen's inequality we can obtain 

V" , ^ P(xi..n) i ST 1 / M P{ x l-n) 

> -u(A) log £^ > -/i(A) log - ~ (13) 



9 



Thus, from flT3J) and (JTOJ) we get 

II > -^\T^)\ogp(T;\T^) - 1/2 > - M (^\2j)loge2 - 1/2. (14) 

Furthermore, 



i \ fi 



xi.. n eX n \T7; 



>-^-„(X n \T;)nlog\X\>~-\og\X\, (15) 



where in the second inequality we have used the fact that entropy is maximized when all events are 
equiprobable, in the third one we used \X n \TJ}\ < \X\ n , while the last inequality follows from ([6]). 
Combining (fTT|) with the bounds (fT2|) . (JTSJ) and (fl~5j) we obtain 

dn(ji,p)>- log n - M(r;\T| ) log £ £ - 1 - log | X \ , 

so that 

M(r;\T^„) < -J—^^^+logn + l + logl^l). (16) 

Since d n (p,,p) = o(n), we can define the parameters e" in such a way that — loge™ = o(n) while 
at the same time the bound (fT6|) gives ^(T™\T J ™ ) = o(l). Fix such a choice of eJJ. Then, using 
/i(T") — ► 1, we can conclude 

p(X n \T%) < K**V?) + M^V^) = (17) 
We proceed with the proof of d n (p, v) = o(n). For any x\„ n S T™ n we have 

n 2n 

where the first inequality follows from Q, the second from (jSJ), and in the equality we have used 
P 2 andi» 



K^l-n) > ^W„Z'n(zi..n) > ^W n Wjn-p(xi.. n ) = (e™) 2 p{xi.. n ) , (18) 



tt>jn = w/(j™) 2 and = 1/en- Next we use the decomposition 



d n (/^)=- M(^l-n) log ^4 - £ ^l..n) log ^4 = / + /J. (19) 

From (|18p we find 

'V, nN2\ / \ i P(xi..n) 



J ii 



t 



(1 + 3 log n - 2 log e" - 2 log w) + 



d n (p, p) + ^ /i(a;i..n) log /> " ' 1 " ' 



\ x!.. n eX n \T% 
< o(n) - ^ M^l-.n) log M^L.n.) 



H.. n 6* n \T5, 



< o(n) + p{X n \T^)n\og \X\ = o(n), (20) 



10 



where in the second inequality we have used — loge™ = o(n) and d n (p, p) = o(n), in the last inequal- 
ity we have again used the fact that the entropy is maximized when all events are equiprobable, 
while the last equality follows from (|17p . Moreover, from ([9]) we find 

JJ< log 2- ]T ^(xi..O log ^^<l + nM(^ n \25) log |Af | =o(n), (21) 

J/J 

where in the last inequality we have used 7(2:1. , n ) = \X\~ n and p(x\„ n ) < 1, and the last equality 
follows from (|17p . 

From flUD, ([20]) and §Z\§ we conclude ±d n (v,p) -> 0. 

Step r: the regularizer 7. It remains to show that the i.i.d. regularizer 7 in the definition of 
v (J9]), can be replaced by a convex combination of a countably many elements from C. Indeed, for 
each n G N, denote 

A n := {xL.n 6r:]/i£C /i(x L . n ) ^ 0}, 

and let for each xi.. n G .Y" the measure p Xl be any measure from C such that p, Xl n (xi__ n ) > 
|sup AteC ^(a;i.. n ). Define 

7n( x 'l..n) := TXT ^ MsBl..n( x l..n)) 
I A n | . 

for each x[ n G A n , n G N, and let 7' := X^fceN w kl'k- ^ or every /i G C we have 

7 / ( a? l..n) > «'n|Air 1 ^a!i.. n (a;i..n) > ^W n \X\~ n n(x 1 „ n ) 

for every ra G N and every x\.. n G A n , which clearly suffices to establish the bound II = o(n) as 
in d2Il). □ 



Example: countable classes of measures. A very simple but rich example of a class C that 
satisfies the conditions of both the theorems above, is any countable family C = {^^ : k G N} 
of measures. In this case, any mixture predictor p := XlfcgN w kHk predicts all fi G C both in 
total variation and in expected average KL divergence. A particular instance that has gained much 
attention in the literature is the family of all computable measures. Although countable, this family 
of processes is rather rich. The problem of predicting all computable measures was introduced in 
[Solomonoff(1978j| where a mixture predictor was proposed. 

Example: Bernoulli i.i.d. processes. Consider the class Cb = {p P ■ p G [0,1]} of all Bernoulli 
i.i.d. processes: fJ, p (xk = 0) = p independently for all k G N. Clearly, this family is uncountable. 
Moreover, each set 

T p := {x G X 00 : the limiting fraction of 0s in x equals p}, 

has probability 1 with respect to p p and probability with respect to any /v : p' ^ p. Since the 
sets T p , p G [0, 1] are non-overlapping, there is no measure p for which p(T p ) > for all p G [0, 1]. 
That is, there is no measure p with respect to which all p p are absolutely continuous. Therefore, by 
Theorem [H a predictor that predicts any p, G Cb in total variation does not exist, demonstrating 
that this notion of prediction is rather strong. However, we know (e.g. [Krichevsky (1993)] ) that 
the Laplace predictor ([T|) predicts every Bernoulli i.i.d. process in expected average KL divergence 
(and not only). Hence, Theorem [2] implies that there is a countable mixture predictor for this 



11 



family too. Let us find such a predictor. Let p q : q G Q be the family of all Bernoulli i.i.d. 
measures with rational probability of 0, and let p := ^2 q& QWqPq, where w q are arbitrary positive 
weights that sum to 1. Let p p be any Bernoulli i.i.d. process. Let h(p,q) denote the divergence 
plog(p/q) + (1 — p) log(l — p/1 — q). For each e we can find a q G Q such that h(p, q) < e. Then 



1 , , x 1 T7 , log/Xp(xi..„) 1 log/X p (>l.. n ) 

n n logp(a;i.. n ) n p w g log p q (xi.. n ) 

l ° gWq +h(p,q) <e + o(l). (22) 



n 

Since this holds for each e we conclude that ^d n (p p , p) — > and p predicts every /j S in expected 
average KL divergence. 

Example: stationary processes. In [Ryabko(19 88) | a predictor pr was constructed which 
predicts every stationary process p G in expected average KL divergence. (This predictor 
is obtained as a mixture of predictors for fe-order Markov sources, for all k G N.) Therefore, 
Theorem [3] implies that there is also a countable mixture predictor for this family of processes. 
Such a predictor can be constructed as follows (the proof in this example is based on the proof in 
Ryabko and Astola(2006)] , Appendix 1). Observe that the family of fc-order stationary binary- 



valued Markov processes is parametrized by 2 [0, l]-valued parameters: probability of observing 
after observing x\,,k, for each x\,± G X k . For each k G N let p q , q G Q 2k be the (countable) 
family of all stationary /c-order Markov processes with rational values of all the parameters. We will 
show that any predictor v := YlkeN Yl q ^Q2 k WkW q p q , where Wk, k G N and w q ,q G Q 2 , k G N are 
any sequences of positive real weights that sum to 1, predicts every stationary p G Cs in expected 
average KL divergence. For p G Cs and k G N define the fc-order conditional Shannon entropy 
hk{p) '■= E M log//(xfc_|_i|xi..fc). We have hk+i(p) > h) c (p) for every k G N and p G C5, and the limit 

hoo(p) ■■= lim h k (p) (23) 

fe^oo 

is called the limit Shannon entropy, see e.g. [Gallager(1968)] . Fix some p G C s . It is easy to see 
that for every e > and every k G N we can find a A:-order stationary Markov measure p qs , q £ G Q 2 
with rational values of the parameters, such that 

^log-^IffW. (24) 

We have 

-d n (p, v) < 2_ + -d n (p,p ) 

n n n He 



0{k/n) + -E„log/i(xi.. n ) E^log/4 (z L . n ) 

1 ™ 

= o(l) + ^too(^) E^^log/^cctlzL.t-i) 



fc=i 



o(l) + h OQ (p) - -E^^log p q£ (x t \xi..t-l) - -B^log Pg^Xk+^XL.k) 

" i=l 



<o(l) + /ioo(^)-^— ^(fc fc (Ai)-e), (25) 
n 
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where the first inequality is derived analogously to (|22|) . the first equality follows from ((3|), the 
second equality follows from the Shannon-McMillan-Breiman theorem (e.g. Gallager (1968) | ) , that 
states that ~ log fi(xi„ n ) — * h 00 (fi) in expectation (and a.s.) for every p € Cs, and ([3]); in the third 
equality we have used the fact that p^ s is fc-order Markov and p is stationary, whereas the last 
inequality follows from (|24p . Finally, since the choice of k and e was arbitrary, from (|25p and (|23p 
we obtain lim n _ i . 00 -d n (p, v) = 0. 

Example: weights may matter. Finally, we provide an example that illustrates the difference 
between the formulations of Theorems [2] and O in the latter the weights are not arbitrary. We will 
construct a sequence of measures u^, k EN, a measure p, and two sequences of positive weights 
and w' k with Y^keN w k = ^keN w 'k = 1> f° r which u := ^^eN^^k^k predicts p in expected average 
KL divergence, but v' := X^fcgN w 'k v k does not. Let Vk be a deterministic measure that first outputs 
k 0s and then only Is, k € N. Let = w/k 2 with w = 6/tt 2 and w' k = 2~ k . Finally, let p be a 
deterministic measure that outputs only 0s. We have d n {p,u) = — log(^fc> n w k) = O(logn), but 
d n (p, v') = - log(^]fe> n w' k ) = - log(2~ n+1 ) = n - 1 / o(n), proving the claim. 



4 Characterizing predictable classes 

Knowing that a mixture of a countable subset gives a predictor if there is one, a notion that 
naturally comes to mind when trying to characterize families of processes for which a predictor 
exists, is separability. Can we say that there is a predictor for a class C of measures if and only if C 
is separable? Of course, to talk about separability we need a suitable topology on the space of all 
measures, or at least on C. If the formulated questions were to have a positive answer, we would 
need a different topology for each of the notions of predictive quality that we consider. Sometimes 
these measures of predictive quality indeed define a nice enough structure of a probability space, but 
sometimes they do not. The question whether there exists a topology on C, separability with respect 
to which is equivalent to the existence of a predictor, is already more vague and less appealing. 
Nonetheless, in the case of total variation distance we obviously have a candidate topology: that of 
total variation distance, and indeed separability with respect to this topology is equivalent to the 
existence of a predictor, as the next theorem shows. This theorem also implies Theorem O thereby 
providing an alternative proof for the latter. In the case of expected average KL divergence the 
situation is different. While one can introduce a topology based on it, separability with respect 
to this topology turns out to be a sufficient but not a necessary condition for the existence of a 
predictor, as is shown in Theorem [5j 

4.1 Separability 

Definition 3 (unconditional total variation distance). Introduce the (unconditional) total variation 
distance 

v(fj,,p) := sup \im(A) - p(A)\. 

Theorem 4. Let C be a set of probability measures on (X 00 ,^). There is a measure p such that p 
predicts every p £ C in total variation if and only if C is separable with respect to the topology of 
total variation distance. In this case any measure v of the form v = Ylh=i w kl i k> where {pf. : k £ N} 
is any dense countable subset of C and Wk are any positive weights that sum to 1, predicts every 
p £ C in total variation. 



13 



Proof. Sufficiency and the mixture predictor. Let C be separable in total variation distance, and 
let V = {i>k : k £ N} be its dense countable subset. We have to show that v := J2kGN w k l/ k, where 
Wk are any positive real weights that sum to 1, predicts every p £ C in total variation. To do this, 
it is enough to show that p(A) > implies v{A) > for every A £ T and every p £ C. Indeed, 
let A be such that p(A) = e > 0. Since V is dense in C, there is a k £ N such that v(p, Vk) < e/2. 
Hence Vk{A) > p{A) — v(p, Vk) > e/2 and v{A) > WkVk(A) > Wk£/2 > 0. 

Necessity. For any p £ C, since p predicts p in total variation, p has a density (Radon- 
Nikodym derivative) / M with respect to p. We can define L\ distance with respect to p as follows 
Ll(p, u) = J Xoo \ ff_i~ f u \dp. The set of all measures that have a density with respect to p is separable 
with respect to this distance (for example a dense countable subset can be constructed based on 
measures whose densities are step- functions with finitely many steps, that take only rational values, 
see e.g. [Kolmogorov and Fomin(1975)| ); therefore, its subset C is also separable. Let V be any 
dense countable subset of C. Thus, for every p £ C and every e there is a p! £ V such that 
< e. For every measurable set A we have 



\p{A)-p'{A)\ 



/ Udp- / fadp < / \fp-fc\dp< / \U~f^\dp< 

J A J A J A Jx°° 



Therefore, v(p,p') = sup A&:F \p(A) — p'(A)\ < e and the set C is separable in total variation 
distance. □ 

Definition 4 (asymptotic KL "distance" D). Define asymptotic expected average KL divergence 
between measures p and p as 

£)(//, p) = lim sup -dn(p,p). (26) 

n^roo Tl 

Theorem 5. For a set C of measures, separability with respect to the asymptotic expected average 
KL divergence D is a sufficient but not a necessary condition for the existence of a predictor: 

(i) If there exists a countable set T> := {v^ : k £ N} C C such that for every p £ C and every 
e > there is a measure p! £ T> such that D(p,p') < e, then every measure v of the form 
v = X^fcLi w kPk, where Wk are any positive weights that sum to 1, predicts every p £ C in 
expected average KL divergence. 

(ii) There is an uncountable set C of measures and a measure v such that v predicts every p £ C 
in expected average KL divergence, but p\ ^ p2 implies D(p\,p2) = oo for every pi,p2 S C; 
in particular, C is not separable with respect to D. 

Proof, (i) Fix p £ C. For every e > pick k £ N such that D(p, Uf.) < £■ We have 



— < top log — 



d n {p,v) = E^log-y— ^ < E M log - = -logwfc + d n (p,v k ) < ne + o{n). 



Since this holds for every e, we conclude ^d n (p, v) — > 0. 

(ii) Let C be the set of all deterministic sequences (measures concentrated on just one sequence) 
such that the number of 0s in the first n symbols is less than y/n. Clearly, this set is uncountable. 
It is easy to check that p\ ^ p% implies D(p\,p2) = oo for every pi,p2 £ C, but the predictor v 
given by v{x n = 0) = 1/n independently for different re, predicts every p £ C in expected average 
KL divergence. □ 
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Examples. Basically, the examples of the preceding section carry over here. Indeed, the example of 
countable families is trivially also an example of separable (with respect to either of the considered 
topologies) family. For Bernoulli i.i.d. and &-order Markov processes, the (countable) sets of 
processes that have rational values of the parameters, considered in the previous section, are dense 
both in the topology of the parametrization and with respect to the asymptotic average divergence 
D. It is also easy to check from the arguments presented in the corresponding example of Section [3] 
that the family of all fc-order stationary Markov processes with rational values of the parameters, 
where we take all k £ N, is dense with respect to D in the set Cs of all stationary processes, so that 
Cs is separable with respect to D. Thus, the sufficient but not necessary condition of separability 
is satisfied in this case. On the other hand, neither of these latter families is separable with respect 
to the topology of total variation distance. 



4.2 Conditions based on the local behaviour of measures. 

Next we provide some sufficient conditions for the existence of a predictor based on local character- 
istics of the class of measures, that is, measures truncated to the first n observations. First of all, 
it must be noted that necessary and sufficient conditions cannot be obtained this way. The basic 
example is that of a family Co of all deterministic sequences that are from some time on. This 
is a countable class of measures which is very easy to predict. Yet the class of measures on X n 
obtained by truncating all measures in Cq to the first n observation coincides with what would be 
obtained by truncating all deterministic measures to the first n observation, the latter class being 
obviously not predictable at all (see also examples below). Nevertheless, considering this kind of 
local behaviour of measures, one can obtain not only sufficient conditions for the existence of a 
predictor, but also rates of convergence of the prediction error. It also gives some ideas of how to 
construct predictors, for the cases when the sufficient conditions obtained are met. 

For a class C of stochastic processes and a sequence x\„ n S X n introduce the coefficients 

°x! „(C) := sup/i(x L . n ). (27) 

Define also the normalizer 

c n (C):= £ c x ,JC). (28) 

Definition 5 (NML estimate). The normalized maximum likelihood (e.g. \Krichevsky(1993Jl ) 
estimator A is defined as 

Ac(zi..n) := — ^^(C), (29) 

CniC) 

for each x\__ n £ X n . 

The family Xc(xi.. n ) (indexed by n) in general does not immediately define a stochastic process 
over X°° (Ac are not consistent for different n); thus, in particular, using average KL divergence 
for measuring prediction quality would not make sense, since 

d n (K-\ x l-n-i), Ae(-|#i.. n -i)) 
can be negative, as the following example shows. 

Example: negative d n for NML estimates. Let the processes /ij, i £ {1, ... ,4} be defined on 
the steps n = 1, 2 as follows. /ii(00) = /i 2 (01) = (^(11) = 1, while /x 3 (01) = /x 3 (00) = 1/2. We have 
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A c (l) = A c (0) = 1/2, while A c (00) = A c (01) = A c (ll) = 1/3. If we define X C (x\y) = X c (yx)/X c (y) 
we get A C (1|0) = A c (0|0) =2/3. Then d 2 (p 3 {-\0), A c (-|0)) = log 3/4 < 0. 

Yet, by taking an appropriate mixture, it is still possible to construct a predictor (a stochastic 
process) based on A, that predicts all the measures in the class. 

Definition 6 (predictor p c ). Let w := 6/-7T 2 and let Wk ■= ^jp. Define a measure p,^ as follows. 
On the first k steps it is defined as Ac, and for n > k it outputs only zeros with probability 1; so, 
Hk{xi..k) = Xc(xi..k) and pk(x n = 0) = 1 for n > k. Define the measure p c as 

oo 

Pc = ^2w k fik- (30) 

k=l 

Thus, we have taken the normalized maximum likelihood estimates X n for each n and continued 
them arbitrarily (actually, by a deterministic sequence) to obtain a sequence of measures on , T) 
that can be summed. 

Theorem 6. For a class C of stochastic processes the predictor p c defined above satisfies 

U nM <^^ + o(^]- (31) 
n n \ n J 

in particular, if 

log c n (C) = o(n). (32) 
then p c predicts every p G C in expected average KL divergence. 
Proof. Indeed, 

-d n (p,p c ) = -Elog — '- < -Elog 



n n Pc(xi.. n ) n w n p n (xi.. n ) 

< Ii og £^£) = I(logc„(C) + 21ogn + log^). (33) 
n w n n 

□ 

Example: i.i.d., finite-memory. To illustrate the applicability of the theorem we first consider 
the class of i.i.d. processes Cb over the binary alphabet X = {0, 1}. It is easy to see that for each 
X\, . . . , x n 

sup p( Xl .. n ) = (k/n) k (l - k/n) n - k 

where k = < n : X{ = 0} is the number of 0s in x±, . . . ,x n . For the constants c n (C) we can 
derive 



c n (C)= SU P V{xi.. n ) = Y. (k/n)\l-k/n) n - k 

Y [I) (k/n) k (l - k/nT~ k < E E Q (W* 1 " = n + L 



fc=0 v ' k=0 t=0 

so that c n (C) < n + 1. 
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In general, for the class Ck of processes with memory k over a finite space X we can get 
polynomial c n (C) (see e.g. [Kr ichevsky (1993)] , and also Ryabko and Hutter(2007)] ), Thus, with 



respect to finite-memory processes, the conditions of Theorem [6] leave ample space for the growth 
of c n (C), since (|32l) allows subexponential growth of c n {C). Moreover, these conditions are tight, as 
the following example shows. 

Example: exponential coefficients are not sufficient. Observe that the condition (|32j) cannot 
be relaxed further, in the sense that exponential coefficients c n are not sufficient for prediction. 
Indeed, for the class of all deterministic processes (that is, each process from the class produces 
some fixed sequence of observations with probability 1) we have c n = 2 n , while obviously for this 
class a predictor does not exist. 

Example: stationary processes. For the set of all stationary processes we can obtain c n (C) > 
2 n /n (as is easy to see by considering periodic n-order Markov processes, for each n 6 N), so 
that the conditions of Theorem [6] are not satisfied. This cannot be fixed, since uniform rates of 
convergence cannot be obtained for this family of processes, as was shown in [Ryabko (1 988)] . 



Optimal rates of convergence. A natural question that arises with respect to the bound (|3ip 
is whether it can be matched by a lower bound. This question is closely related to the optimality of 
the normalized maximum likelihood estimates used in the construction of the predictor. In general, 
since NML estimates are not optimal, neither are the rates of convergence in (|3ip . To obtain (close 
to) optimal rates one has to consider a different measure of capacity. 

To do so, we make the following connection to a problem in information theory. Let ^(X 00 ) be 
the set of all stochastic processes (probability measures) on the space (X 00 ,^), and let V(X) be 
the set of probability distributions over a (finite) set X. For a class C of measures we are interested 
in a predictor that has a small (or minimal) worst-case (with respect to the class C) probability of 
error. Thus, we are interested in the quantity 

inf sup D (it, p), (34) 

where the infimum is taken over all stochastic processes p, and D is the asymptotic expected average 
KL divergence (126j) . (In particular, we are interested in the conditions under which the quantity (|34p 
equals zero.) This problem has been studied for the case when the probability measures are over 
a finite set X, and D is replaced simply by the KL divergence d between the measures. Thus, 
the problem was to find the probability measure p (if it exists) on which the following minimax is 
attained 

R(A):= inf sup d(p,p), (35) 
pev(x) 

where A C V(X). This problem is closely related to the problem of finding the best code for 
the class of sources A, which was its original motivation. The normalized maximum likelihood 
distribution considered above does not in general lead to the optimum solution for this problem. 
The optimum solution is obtained through the result that relates the minimax (|35p to the so-called 
channel capacity. 

Definition 7 (Channel capacity). For a set A of measures on a finite set X the channel capacity 
of A is defined as 

C{A):= sup V P(p)d(p,p P ), (36) 
p ^ A ^eS(P) 

where Vq{A) is the set of all probability distributions on A that have a finite support, S(P) is the 
(finite) support of a distribution P G Vq(A), and pp = X^eS(P) -^(a 4 )/- 4 - 
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It is shown in [Ryabko(1979)| |Gallager(1976 (revised 1979))] that C(A) = R(A), thus reducing 



the problem of finding a minimax to an optimization problem. For probability measures over 
infinite spaces this result (R(A) = C(A)) was generalized by [Haussler(1997)| , but the divergence 
between probability distributions is measured by KL divergence (and not asymptotic average KL 
divergence), which gives infinite R{A) e.g. already for the class of i.i.d. processes. 

However, truncating measures in a class C to the first n observations, we can use the results 
about channel capacity to analyze the predictive properties of the class. Moreover, the rates 
of convergence that can be obtained along these lines are close to optimal. In order to pass from 
measures minimizing the divergence for each individual n to a process that minimizes the divergence 
for all n we use the same idea as when constructing the process p c . 

Theorem 7. Let C be a set of measures on (X 00 ,^), and let C n be the class of measures from C 
restricted to X n . There exists a measure pc such that 

X A( (\ogn\ 

-d n {p,p c )< + 0[ ; (37) 

n n \ n J 

in particular, if C(C n )/n — ► then pc predicts every p G C in expected average KL divergence. 
Moreover, for any measure pc and every e > there exists p £ C such that 

1 , . . C(C n ) 

-d n (p,pc) > e. 

n n 



Proof. As shown in Gallager(1976 (revised 1979))] , for each n there exists a sequence v-£, k G N of 



measures on X n such that 



lim sup d n (p,v%) -► C{C n ). 



For each n G N find an index k n such that 

| sup d n (fjL,v% n )-C(C n )\ < 1. 
nee n 

Define the measure p n as follows. On the first n symbols it coincides with and Pn(x m = 0) = 1 
for m > n. Finally, set pc = Y^=i w nPn, where Wk = -^s,w = 6/tt 2 . We have to show that 
limn^oo ~d n (p,,pc) = for every p G C. Indeed, similarly to (f33|) . we have 

-d n {p,p c ) = r E M log. 



n n Pc{xi..n) 



logu; fc 1 p(xi.. n ) .log w + 2 log n 1 

< — + -E„ log — < h -d n (p, p ri 

n n Pn(xi..n) n n 



< (1) + ^). (38) 
n 



The second statement follows from the fact |Ryabko(1979) Gallager(1976 (revised 1979))| that 



C{C n ) = R{C n ) (cf. ([35|)). □ 

Thus, if the channel capacity C(C n ) grows sublinearly, a predictor can be constructed for the 
class of processes C. In this case the problem of constructing the predictor is reduced to finding 
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the channel capacities for different n and finding the corresponding measures on which they are 
attained or approached. 

Examples. For the class of all Bernoulli i.i.d. processes, the channel capacity C(C^) is known to 
be O(logn) |Krichevsky(1993)] . For the family of all stationary processes it is 0(n), so that the 
conditions of Theorem [7] are satisfied for the former but not for the latter. 

We also remark that the requirement of a sublinear channel capacity cannot be relaxed, in the 
sense that a linear channel capacity is not sufficient for prediction, since it is the maximal possible 
capacity for a set of measures on X n , achieved e.g. on the set of all measures, or on the set of all 
deterministic sequences. 



5 Discussion 



The first possible extension of the results of the paper that comes to mind is to find out whether 
the same holds for other measures of performance, such as prediction in KL divergence without 
time-averaging, or with probability 1 rather then in expectation, or with respect to other measures 
of prediction error, such as absolute distance. (See Ryabko and Hutter(2007)] for a discussion of 
different measures of performance and relations between them.) Maybe the same results can be 
obtained in more general formulations, for example, using /-divergences of [Csiszar ( 1967)1 . 

More generally, the questions we addressed in this work are a part of a larger problem: given 
an arbitrary class C of stochastic processes, find the best predictor for it. We have considered two 
subproblems: first, in which form to look for a predictor if one exists. Here we have shown that 
if any predictor works then a Bayesian one works too. The second one is to characterize families 
of processes for which a predictor exists. Here we have analyzed what the notion of separability 
furnishes in this respect, as well as identified some simple sufficient conditions based on the local 
behaviour of measures in the class. Another approach would be to identify the conditions which two 
measures p and p have to satisfy in order for p to predict p. For prediction in total variation such 
conditions have been identified Blackwell and Dubins(1962) , Kalai and Lehrer(1994) and, in par- 
ticular, in the context of the present work, they turn out to be very useful. Kalai and Lehrer(1994)| 
also provide some characterization for the case of a weaker notion of prediction: difference be- 
tween conditional probabilities of the next (several) outcomes (weak merging of opinions). In 
Ryabko and Hutter(2008b)] some sufficient conditions are found for the case of prediction in ex- 



pected average KL divergence, and prediction in average KL divergence with probability 1. Of 
course, another very natural approach to the general problem posed above is to try and find pre- 
dictors (in the form of algorithms) for some particular classes of processes which are of practical 
interest. Towards this end, we have found a rather simple form that some solution to this question 
has if a solution exists: a Bayesian predictor whose prior is concentrated on a countable set. We 
have also identified some sufficient conditions under which a predictor can actually be constructed 
(e.g. using NML estimates). However, the larger question of how to construct an optimal predictor 
for an arbitrary given family of processes, remains open. 

Taking an even more general perspective, one can consider the problem of finding the best 
response to the actions of a (stochastic) environment, which itself responds to the actions of a 
learner. Allowing into consideration environments that change their behaviour in response to the 
action of the learner, clearly makes the problem much more difficult, but it also dramatically 
extends the range of applications. For this general problem one can pose the same questions: given 
a set C of environments, how can we construct a learner that is (asymptotically) optimal if any 
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environment from C is chosen to generate the data? One can consider Bayesian learners for this 
formulation too [Hutter(2005)| ; it would be interesting to find out whether one can show that when 
there is an learner which is optimal in every environment from C, then there is a Bayesian learner 
with a countably supported prior that has this property too. 
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